Using augmented reality in surgical navigation

ABSTRACT

A surgical navigation system may include a processor and a display. The processor may receive a patient image and sensor data captured by a sensor, receive a medical image, generate a hologram of the medical image, perform coregistration between the patient image and the hologram, superimpose the hologram on the patient image, and display the superimposed image. Coregistration may be performed manually via a user interaction, or automatically based on one or more fiducials in the medical image and sensor data related to the fiducials. The system may monitor a change in the environment and update the display correspondingly. For example, the system may monitor a movement of a body of the patient, monitor the size of an organ of the patient as the organ is being under operation, or a movement of the surgical instrument. The sensor may be an augmented reality (AR) sensor in an AR device.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims benefit of priority pursuant to 35 U.S.C. §119(e) of U.S. provisional patent application No. 62/488,452 entitled“SYSTEM, METHOD AND COMPUTER-ACCESSIBLE MEDIUM FOR THE USE OF AUGMENTEDREALITY FOR THE SURGICAL NAVIGATION,” filed Apr. 21, 2017, which ishereby incorporated by reference in its entirety.

FIELD OF THE DISCLOSURE

The present disclosure relates generally to surgical navigation, andexamples of using augmented reality (AR) during medical procedures aredisclosed herein.

BACKGROUND

A major challenge during a surgery is to differentiate between diseasedtissues and healthy tissue. Traditional procedures in a surgery requirethat a surgeon visually inspect and grossly determine whether aninvolved tissue is different from a healthy one. Current neuronavigationsystems take an image of a patient's body part prior to the surgery,display the image on a screen during a surgery, and correlate asurgeon's instrument to a location of the image in real-time. However,neuronavigation systems require a surgeon to look away from the patientduring the operation. As such, the existing systems present technicalproblems.

SUMMARY

A system using augmented reality (“AR”) for surgical navigation mayreceive medical image of interest, such as from medical resonanceimaging (“MRI”), computed tomography (“CT”), ultrasound (“US”),microscopes, or any other devices. The system may use a sensor, e.g., anAR sensor, to detect changes in the environment, render hologramsrepresenting the medical image, and place holograms relative to theenvironment, e.g., a patient's body.

Various different devices can be used as the sensor forAR-neuronavigation system. The examples of sensors include ultrasound, acamera (video, SLR, infrared etc.), or any other 3D scanning device.Multiple such devices of one type or multiple types might be used toincrease the accuracy during procedures, surgery or other medicalinterventions. The placement of the images relative to the patient'sbody can be achieved using coregistration. The coregistration usesinformation from image of interest and the environment and then thesystem may use this mutual information to place the image of interestrelative to the environment. The system may display the structures ofinterest from the image as a hologram in the real world, or displayimages and objects of interest on to a screen.

The coregistration can be accomplished in multiple ways. For example,the system may use the holographic rendering of patient's skin visibleon medical image of interest and correlate that with the actual skinsensed by the AR system. The system may also allow a user to adjust theholographic rendering of the skin relative to the patient's bodymanually. Additional fiducials can be placed on patient's body, thefiducials may be visible on the medical image or can be sensed by the ARsensors. These fiducials can then guide the placement of the hologramrelative to the patient. The system may also use 3D scanning as an ARsensor, and the resulting information can be correlated with the imageof interest, which allows the accurate placement of the hologramsrelative to the patient body.

Additionally, and/or alternatively, the system may display a magnifiedview of the areas of interest by gathering high definition images and orcombining multiple modalities, and creating magnified holograms. Thismay facilitate precise surgery. For example, the system may provide abinocular view in cases where it is otherwise impossible with othermeans, e.g. endoscopic, or laparoscopic surgery. Different objects,organs, lesions, or other areas of interest can be shaded or coloreddifferently to further help easier identification.

The information from the sensors can be used to perform thecoregistration as described above globally or locally. For example, inaddition to global coregistration, an ultrasound probe can be insertedinto the body to provide better and more precise local information. Thislocal information can be used as is or can also be overlaid on to thepreviously existing global coregistration.

In some examples, the system may track fiducials that are placed on apatient's face, for example, and update the coregistration according tothe displacement and or rotation of the fiducials. This allows a surgeryto be performed without requiring the patient to have the patient's bodypart, e.g., the head, immobilized by placing it in pins (Mayfield).Similar results can also be achieved by using facial recognition methodsinstead of using fiducials, where natural facial features serve asfiducials.

The system provided herein can be used as intra operative imagingdevice. For example, the system may detect changes in the surgicalenvironment in real time, and update the representation of the realworld. This updated representation of the real world can be overlaidonto an MRI, and can help assess, for example, the amount of surgicalprogress, such as the amount of tumor that has been removed.

In some examples, the system may update the map and structure of objectsin its surroundings at regular, desired, or custom intervals. Forexample, in surgery, a nerve is moved to a new location due tomanipulation, and the system may detect the movement of the nerve andre-arrange the holographic representation beyond the initial medicalimage to reflect the updated location of the nerve.

This rearrangement can be further projected onto an initial medicalimage, and the initial medical image can be updated to reflect thecurrent situation. Hence, as one non-limiting example, a new MRI will becreated reflecting current anatomy, using information from devices beingused as AR sensors without requiring patient to have another MRI.

The system may also detect changes in the internal body organs. Forexample, during neurosurgery, the brain can become edematous. The systemmay detect a change in the size of the brain, for example, and correlatethe changed size of the brain to the previously received medical image.Hence, brain edema can be quantified. The system may also detect bloodloss during surgery. Similarly, image processing and updates in objectshapes can help inform surgeons and other medical staff about real timecardiac output and lung function during cardiac surgery. The examplesused hereinabove, which include the brain, blood, heart and lung, havebeen provided as representative examples and do not limit the scope ofthe disclosure. For example, the system described herein may also applyto other body organs.

In some scenarios, the system may detect the hands of surgical ormedical personnel, as well as any instruments used in surgery, via oneor more AR sensors, such as 3D scanners. The hand(s) and/or instrumentscan be then mapped and displayed on to the MRI image, or on theholograms. This is advantageous for many reasons. For example, it caneliminate or reduce the need for special probes. In order to enhance thesensitivity of instruments or personnel hands to the 3D scanning devicebeing used, they may be coated with special materials to allow easiermapping. As a non-limiting example, a surgeon's hands can be made moresensitive by coating the gloves with any material that increasessensitivity. Special pointers with easier to detect materials built intothem can also be used to allow surgeons to point to a structure onpatient, which will then map the pointer onto the image or hologram.

BRIEF DESCRIPTION OF THE DRAWINGS

Further objects, features and advantages of the present disclosure willbecome apparent from the following detailed description taken inconjunction with the accompanying Figures showing illustrativeembodiments of the present disclosure, in which:

FIG. 1 illustrates a surgical navigation system in accordance withexamples described herein.

FIG. 2 illustrates examples of processes for using augmented reality insurgical navigation in accordance with examples described herein.

FIG. 3 illustrates an example of a process for manual coregistration inaccordance with examples described herein.

FIGS. 4A-4B illustrate the overlay of the skin of a patient's head andthe brain over a patient image in accordance to some examples describedherein.

FIGS. 5A-5D illustrate an example of surgical navigation.

Throughout the drawings, the same reference numerals and characters,unless otherwise stated, are used to denote like features, elements,components or portions of the illustrated embodiments. Moreover, whilethe present disclosure will now be described in detail with reference tothe figures, it is done so in connection with the illustrativeembodiments and is not limited by the particular embodiments illustratedin the figures and the appended claims.

DETAILED DESCRIPTION

In FIG. 1, a surgical system 100 may include an augmented reality (AR)device 104. The AR device 104 may include one or more AR sensors 106, adisplay 108, a communication port 110, and a processor 112. The ARsensor 106 is a device or a combination of devices that may record anddetect changes in the environment, e.g., a surgery or procedure room 102having a patient. These devices may include cameras (e.g. infrared, SLR,etc.), fiducials placed at known places, ultrasound probes, or any otherdevice that is capable of three-dimensional (3D) scanning, to captureinformation and images in the environment. The AR sensor may also be anAR sensing device. The display 108 may display images captured from theAR sensor(s) or other sensors. Display 108 may be capable of renderingholograms. For example, the AR device 104 may include a goggle, in whichthe rendering of holograms can be done in real world or on a lens.Holograms can also be displayed as masks on top of images on a screen.Processor 112 may execute programming instructions to perform certainfunctions. For example, the programming instructions, when executed, maycause the processor to superimpose holograms relative to the real worldconstructed from the AR sensor(s) 106 and make them appear to be in acertain relationship to real world objects, such as a body, e.g., thehead of a patient.

System 100 may also include one or more imaging modalities 116 that areconfigured to capture medical images 114. The imaging modalities mayinclude, for example, MRI, CT scan, ultrasound, a microscope, or anyother device. The medical images can be prerecorded or can becontinuously obtained in real time. The medical images may include astatic image or a sequence of images over time (e.g. functional MRI).

In some scenarios, the programming instructions for the processor 112may be configured to coregister the medical image of interest to theinformation, such as a patient image, from the AR sensor, to provide aspatial mapping between the medical image and the patient image. Theprocessor may perform the coregistration automatically by correlatingone or more features of the medical image to the sensor data from the ARsensor. Alternatively, the processor may perform the coregistration in amanual process based on user interactions. Examples of coregistrationwill be further described in this document.

Once the medical image and the patient image are coregistered, thesystem may superimpose a representation of the medical image onto thepatient image. For example, the representation of the medical image maybe a hologram. The representation may also be the medical image itself,or a 3D model constructed from one or more medical images, such as CTimages. The system may display the superimposed medical image and thepatient image altogether in the display 108. This display willfacilitate a view of the medical image in the context of a real-timeenvironment. As one non-limiting example, the medical image may be theCT image of a head, and the system may construct a hologram of apatient's brain and superimpose the hologram onto a real-time image thatincludes a patient's head. An example is shown in FIG. 4B.

Returning to FIG. 1, in some scenarios, coregistration may not be neededif a specific relationship between real world objects and arepresentation of the medical image is not needed. For example, inteaching or surgical planning, images do not need to be overlaid exactlyon to the patient's head. In some scenarios, when a magnified view ofobjects is desired (e.g. during surgery), coregistration may not beneeded either.

In some scenarios, the sensors of the AR device may also capture medicalimages in the surgical AR system. For example, an AR sensor may be anultrasound that can be used to obtain the images from the patient. TheAR sensor images can serve multiple purposes. For example, the AR sensorimages can serve as medical image of interest. The AR sensor images mayalso deliver the information for the AR device for spatial mapping. Insome non-limiting scenarios, the AR sensor may also capture data relatedto the patient image. For example, the AR sensor data may include facesand vertices that describe a two-dimensional (2D) surface in 3D space,or other data pertinent to the other fiducials on the patient's bodypart or system.

Various methods may be implemented in above described system. In FIG. 2,examples of processes for superimposing a representation of a medicalimage onto a patient image are further described. In some scenarios, amethod may start with receiving AR sensor data 202 from one or more ARsensors. AR sensor data may include patient image and/or sensor datarelated to the patient image. The method may also include receivingmedical image 206 from one or more imaging modalities, and performingcoregistration 210 to generate a spatial mapping between the medicalimage and the patient image. In preparing for registration, the methodmay generate volumetric data 204 based on the data from the AR sensor,for example, faces and vertices that describe a two-dimensional (2D)surface in 3D space. The methods may convert a 2D surface into a 3Dmatrix. For example, William E. Lorensen, Harvey E. Cline: MarchingCubes: A high resolution 3D surface construction algorithm. In: ComputerGraphics, Vol. 21, Nr. 4, Jul. 1987, describes a marching cubesalgorithm. Other algorithms may also be available.

In some non-limiting scenarios, the method may further includegenerating a representation of the medical image 207, such as ahologram. For example, the method may generate a hologram of the skin(or any external visible surface) from the CT scan. The skin or anyexternal visible surface of a patient may be suitable as fiducials forcoregistration. In some scenarios, the method may include selecting allthe voxels in the CT scan with an attenuation value of approximately−150 Hu. This will yield a mask with all the voxels other than thosepertaining to patient's head excluded. Optionally, this mask may haveholes in it. For example, the nasogastric structures, nasal sinuses,external ear canals and other structures that normally contain air mayhave the voxels corresponding to air in them excluded as well.Optionally, the method may fill these holes to yield an improvedreconstruction of the head.

In some scenarios, the method may fill the holes by doing a 3D operationon the image. Alternatively, and/or additionally, the method may fillthe holes by doing a 2D operation on each slice of the image. Methodsfor filling holes are known. For example, a sample algorithm for fillingholes is described in Soille, P., Morphological Image Analysis:Principles and Applications, Springer-Verlag, 1999, pp. 173-174. In somescenarios, the method may exclude some small areas that are not part ofa patient's body, e.g., the head, by retaining the connected componentsof the image that exceed a threshold size. In some scenarios, the methodmay receive input from the user to adjust the mask.

Once the mask of fiducials has been segmented, the volumetric data canbe converted into vertices and faces. Many different algorithms can beused for this process as well, one representative one is William E.Lorensen, Harvey E. Cline: Marching Cubes: A high resolution 3D surfaceconstruction algorithm. In: Computer Graphics, Vol. 21, Nr. 4, Jul.1987. This data can be used to create the hologram of the object.

In some scenarios, the method may also generate representations, e.g.,holograms of the objects of interest other than the ones used asfiducials. As one non-limiting example, the method may generateholograms of a patient's body parts, conditions, or malformations,including but not limited to the brain, tumor(s), artery or arteries,vein(s) or hematoma. The relationship between fiducials and otherobjects of interest can be computed using either AR sensor or medicalimage. After the coregistration, the representation can be switched fromfiducials view (e.g. skin view) to object of interest view (e.g. brainand hematoma view).

In some scenarios, the method may include receiving surgical incisionsites and/or trajectories relative to the medical image. The system mayconvert the incisions and trajectories into holograms or representationsof their own, and display the representations of the incisions andtrajectories during the surgery when needed to guide actual incision ortrajectory. In some examples, the holograms are displayed on thepatient. In other examples, the rendering of holograms can be done inreal world or on a lens, such as on goggles.

In some scenarios, the method may include extracting one or morefeatures 208 from the medical image, where the features may be suitablefor use as fiducials in the registration 210. For example, the medicalimage may include a CT head image, and the method may extract the skin(or other structure being used as fiducial). In the case of skin, thiscan be accomplished by the same process that was described withreference to box 207. The method may further discard other unrelatedstructures to allow for better coregistration. In some scenarios, block208 may be option; whereas in other cases, block 208 may help improvethe accuracy of coregistration.

With further reference to FIG. 2, coregistration (block 210) isdescribed in detail. In performing coregistration 210, the method mayneed the medical image (for example, CT head image) to coregister andalso the data from the AR sensor regarding the head of the patient.Existing registration algorithms may be available. For example,intensity based coregistration may be available. Feature basedcoregistration algorithms may also be available.

Additionally, and optionally, the method may receive spatial referencinginformation about both the patient image and the medical image, whichmake the registration process faster. In some scenarios, the spatialreferencing information is important, for example, when the voxels in CThead are not isotropic, with their thickness is always greater thanwidth and length. In AR sensor data, however, voxels are isotropic. Thespatial referencing will help ease this limitation.

In some scenarios, for example, when both images come from one realworld object i.e. patient's head, that was first scanned using CT scanand now is being sensed using AR device, the method may useEuclidian/rigid body registration/registration with six degrees offreedom for block 210.

The method may further generate a transformation matrix 212 which can beused in conjunction with the location of the patient's head from ARsensor data to place the hologram. While the steps in box 210 and 212can be computational expensive, the method may be implemented in acomputer that may be in communication with the AR device. For example,the computer may be external to the AR device and may be connected tothe AR device using USB, Bluetooth, WiFi or other communicationprotocols.

In performing the coregistration 210, the method may use suitablefeatures as fiducials, such as the skin or external surface when the ARsensor includes a camera. Alternatively, and/or additionally, when otherAR sensors, e.g., ultrasound, are used, the method may select otherstructures, e.g. skull for coregistration. In some scenarios, inaddition to using skin as a feature for automatic coregistration, themethod may also place fiducials on the patient's body. For example,fiducials can be placed on the patient's body before acquiring themedical image. This way, the fiducial information in a medical image canbe used in to correlate the medical image with the patient's actualhead. The fiducials can also be placed after the acquisition of medicalimages, in which case they can be detected by the AR sensor device(s).This would facilitate the detection of a change in the environment, suchas a movement of a patient's head. In some scenarios, the method may useany other method of 3D scanning to correlate medical image with thepatient's body.

With reference to FIG. 3, the registration method may include a manualprocess via a user interaction. For example, the method may includereceiving a patient image and AR sensor data 302, receiving a medicalimage 304, and displaying the patient image and a representation of themedical image in the AR device 306. For example, the method may generatea hologram of the external surface or skin of the patient based on themedical image, as described early in this document. The method mayrender the hologram of the external surface/skin relative to theenvironment on the display device of the AR device (e.g. 108 in FIG. 1).The method may initialize at some point relative to the environment,then may allow the user to move and rotate the hologram along x, y or zaxis and receive a user interaction to overlay the patient image and thehologram of the medical image until the user is satisfied with theoverlay result. The method may display superimposed image 310 based onthe user interaction.

Returning to FIG. 2, once the registration is complete, the method maysuperimpose the representation of the medical image onto the patientimage 214 based on the transformation matrix and display thesuperimposed image 226 on the display of the AR device (e.g., 108 inFIG. 1). This may allow a surgeon to directly look at the medical imagein the context of the real object, such as the patient's headsimultaneously, while performing the surgery.

The position, size and the orientation of the hologram is determined bythe values of x, y and z coordinates, rotation and scale. Afterinitialization, the user may view the hologram and the patient's facethrough the AR device. User can move the hologram by changing the valuesfor rotation, scale or location of x, y and z components. This needs tobe done continuously until the user is satisfied with the overlay of thehologram on to the actual patient's skin. In this process, the data fromthe AR sensor regarding the shape of the head is not needed. Instead,the user is looking at the patient's head in the display while movingthe hologram so that it gets overlaid on to the patient's headappropriately.

Instead of displaying a patient's skin using an AR rendering device, themethod can also display on the screen or any other display modality, andthis can allow the user to see the relationship between the real worldpatient body and the skin mask segmented earlier. This can then in turnbe used to help manually coregister the two, if desired.

In FIG. 2, some or all of the devices serving as AR sensors or to obtainthe medical image can be inside the patient body, for example, anendovascular catheter with ultrasound probe on its tip can be placedinside a blood vessel during the procedure and the data obtained fromthe probe can be both used as medial image and a way to detect a changein the environment, which will be described as below.

With further reference to FIG. 2, the method may allow a patient's bodyto move while the surgery is in operation. This will free the patientfrom being immobilized, such as constrained by any pins, e.g., Mayfieldskull pin. In some scenarios, the method may detect a change in theenvironment 216, for example, a movement of the patient's head. Themethod may use data from multiple different devices, such as cameras inthe AR sensor to detect the head movement. The method may also useimages from multiple ultrasound probes and from multiple video camerasto register them together to increase the resolution of the finalrepresentation of the environment. In some scenarios, the method may usefacial recognition to detect the movement of a patient's head during thesurgery. Similarly the method may use 3D scanning, such as using an ARsensing device, to detect the changes in the environment.

In some scenarios, the method may determine whether a change in theenvironment, e.g., the movement of the patient's head, exceeds athreshold T 218. For example, the method may use object recognition totrack a patient's head and provide the updated position and rotation ofthe patient's head. The tracking of an object may be done by existingmethods, such as the methods provided by Vuforia library(https://library.vuforia.com/articles/Solution/How-To-Use-Object-Recognition-in-Unity).If the method determines that the change in the environment relative tothe previous position has exceeded a threshold, the method may determinethe motion information 222. For example, the method may determine thatthe movement of the patient's head has exceeded 1 mm, or the patient'shead has rotated more than one degree.

Once a change in the environment is detected, e.g., a movement of thepatient's head, the method may update the transformation matrix 224. Forexample, the method may obtain the x, y, z rotation and translationcomponents of the transformation matrix, then adding to those componentsthe change in value (obtained in box 222) to update the transformationmatrix 224. The method may further repeat box 214 and box 226, withoutrepeating coregistration 210. As such, the initial coregistration can bemanual, such as shown in FIG. 3, without sacrificing the performance ofthe system.

Alternatively, and/or additionally, the method may receive an updatedpatient image 220 after determining that a change in the environment hasoccurred and/or has exceeded a threshold T 218. For example, the methodmay obtain the entire isosurface mesh for the external surface of thepatient head. The method may repeat boxes 204, 210, 212, 214 and 226. Inother words, the method may repeat coregistration each time a change inthe environment, or a change in the location of fiducials is detected.

The various embodiments in FIG. 2 use continuous updates from one ormore AR sensors to detect a change in the environment, includinganything being used as fiducials, e.g. the skin or external body surfaceof the patient, or custom fiducials placed on/in patient's body. Theupdated location of the fiducials can be correlated with the image ofinterest. Hence the representation (e.g. hologram) of the image ofinterest will move with the moving patient's head. This can helpincrease patient comfort and creating more room for surgery byeliminating the device used for immobilization.

In some scenarios, the methods described in FIG. 2 may facilitate intraoperative monitoring. For example, the system may use the AR sensor todetect changes in the positioning of the normal anatomical structures.For example, during acoustic neuroma resection surgery, theidentification of facial nerve (VII cranial nerve) is of key importance.The nerve can be initially located by image guidance because the anatomymatches the imaging. As drilling of the temporal bone proceeds, thenerve is no longer held in its position. Images can no longer correctlyidentify the nerve because the nerve is at a different location comparedto when image was taken and anatomy has changed. The system may use theAR sensor to continuously detect and update the anatomy as drilling isbeing done. The system may detect the changes in the nerve location andmove the hologram of the nerve as the nerve moves.

In some scenarios, the system may also use the updated knowledge of thenerve to update the initial CT or MRI image that was being used. Hence aCT or MRI image with updated nerve location will be available based onobject tracking performed by the AR sensing device without acquiring anew MRI image.

In some scenarios, the methods described in FIG. 2 may also track thechanges in the anatomy as it is being modified. For example, during anacoustic neuroma surgery, as a bone is being removed, the system maydetect the removal of the bone and update the representation of the bone(whether on a display or as a hologram). This can further in turn beused to update the CT or MRI image with the appropriate portions of thebone removed. The same can be applied to tumor resection. At times,tumor appears as normal to the human eye, but is visibly different onimaging. The updated size of tumor detected by AR-sensor can be overlaidon the images and hence the clinical decision of whether to continue theremoval of the tumor or not can be augmented. It should be noted thatfor intra operative imaging, the removal of acoustic neuroma is beingpresented as an example only. The system may apply to different medicalprocedures on different parts of a body in a similar manner.

In some scenarios, the constant monitoring from AR sensing device canalso be used to quantify a change in anatomical structures. For example,if the brain is getting edematous during surgery, the AR-sensor canquantify the changed brain volume and estimate the edema. Similarly, thesystem may quantify the blood loss in the surgery by continuouslyupdating representations of the environment and output the estimate ofthe blood loss to the user. In some or other scenarios, the system mayquantify a change in heart and/or lung volumes during the cardiac andrespiratory cycles and in turn measure their function.

In some scenarios, the system may use the AR sensor, for example, totrack and capture a movement of a surgeon's hand(s) and instruments. Thesystem may track the location of the surgeon's hands and instruments andoverlay them to the images and holograms. This will allow the user tocorrelate the location of the instrument with the anatomy without usinga special probe. Additionally, special probes may also be used.Optionally, the gloves or the instruments may be coated with a materialthat is easier for AR sensing device to detect. This can in turn, allowthe representation of the instruments or hands to be overlaid on to theimage.

If the AR sensor (e.g., ultrasound) is capable of detecting changes inthe deeper layers of the tissue, then the system may use the AR sensorto find the location of the surgical instruments inside the tissue aswell. Even though static representations of the instrument can beprojected on to the images as well, at times, more flexible cathetersand other instruments e.g., deep brain stimulator leads get bent whilegoing through the brain parenchyma. The system may detect this bentinside the brain by using an ultrasound probe and superimpose it on toimage, which may show to the surgeon the final path and location of thecatheter or deep brain stimulator leads.

It may be appreciated that the boxes shown in FIG. 2 may havevariations, and some may be optional or combined. In some scenarios, thesystem may generate the representation of the medical image 207, e.g., ahologram, and display the hologram without superimposing therepresentation onto the patient image. For example, the system mayprovide a magnified 3D view or a binocular 3D view based on theholograms. In some scenarios, for example, in endoscopic surgery, thecamera provides a 2-D image of the surgical field, and the depth isdifficult for a surgeon to appreciate on the screen. By displaying a 3Dbinocular view or the magnified 3D view, the system may facilitate thesurgeon to better understand the environment and/or see details of thestructures of the patient's body.

Other variations are described herein. In some scenarios, while cameraon the endoscope/laparoscope/bronchoscope can provide the visible view,the system may include additional AR sensors (106 in FIG. 1), such as anultrasound probe that can being used as an AR sensor on the tip of theendoscope, to provide a 3D view. This can be useful in the situationswhere for example blood can obscure the camera view, but in anultrasound image, the surgeon will still be able to view the structurescovered by the blood. When bleeding occurs in a surgery, the system mayallow the surgeon to identify the artery responsible for the bleeding,e.g., in a hologram based on the ultrasound image, and control thebleeding by clamping that artery. This may not be possible by using acamera as the AR sensor when the camera view is obscured by the blood.

Holograms of different organs can be color coded or can be created fromdifferent materials. This difference in shading, colors, transparency orreflection may allow the user to easily differentiate between differenttissue types e.g. lesion versus normal tissues versus blood.

In some scenarios, the system may perform the coregistration (e.g. box210 in FIG. 2) for the entire body part for both the medical image andthe patient image. Alternatively, and/or additionally, the method mayperform a local coregistration that may be suitable for the surgicalfield. For example, the system may use ultrasound probes in the surgicalfield to create a 3D view of the field itself and the structures beneathit, for example, an artery. The method may perform a localcoregistration by correlating the location of an artery in theultrasound from an AR sensing device to the medical image from animaging modality. This will facilitate more precise location of lesionthan possible with global coregistration only.

In some scenarios, the system may also perform a local coregistrationusing local landmarks. These landmarks may be custom, as picked bysurgeons. Local coregistration may be done in a similar manner asdescribed above in various embodiments in FIG. 2. For example, themethod may allow a user to perform a manual coregistration by moving thelocal hologram to be superimposed on the patient image. Any suitableanatomical features may be thought of as natural fiducials and may beused in coregistration. Local coregistration may be advantageous whentissue deformation decreases the accuracy of the projection from globalcoregistration.

Once the coregistration has been done, the hologram of the skin (or theexternal surface, the artery, or any fiducials used etc.) can beswitched to the view of interest e.g. view of surgical incision site andtrajectory, hematoma and or brain.

Various embodiments described herein may facilitate a number of surgicalprocedures. For example, FIG. 4A shows the overlay of a patient's headskin on top of patient's own actual head image. This illustrates theprincipal of using natural landmarks from the patient's body asfiducials. This placement can be achieved manually or automatically asdescribed herein. Once the hologram fully and accurately covers thepatient's head, the view can be switched to the object of interest, forexample, the brain in FIG. 4B.

In a non-limiting example in FIGS. 5A and 5B, the hologram of apatient's head is shown. The head hologram will be moved until itaccurately overlays on to the patient's actual head in all threedimensions. FIGS. 5C and 5D show the subdural hematoma in dark and brainin light. This is the ‘object of interest’ in this example. Therelationship between the object of interest and fiducial is known fromthe medical image being used. 502, 504 are different views of thepatient's head external surface hologram generated using a CT image.506, 512 are the location that surgeon decided to incise. 508 is thelesion (in this example, subdural hematoma) to be drained. 510 is thebrain.

Various embodiments described herein provide solutions to the technicalproblems that exist in prior art systems and are advantageous in helpingsurgeons determine their target easily without looking away from thepatient. The present disclosure also facilitates intra-operative imagingin that the system may detect changes in real-world object shapes anduse information about that change to assess how much the diseasedtissue, e.g., a tumor is left over. This avoids having to take thepatient to the MRI, re-image and compare it to the prior MRI todetermine how much residual is left over, which process is costly andtime consuming.

It will be appreciated that various modifications and alterations to thedescribed embodiments may be possible as one may be able to devisenumerous systems, arrangements, and procedures which, although notexplicitly shown or described herein. For example, multiple medicalimages may be used for coregistration. Various different exemplaryembodiments can be used together with one another, as well asinterchangeably therewith, as should be understood by those havingordinary skill in the art.

In an aspect of the disclosure, a system includes a processor, adisplay, and a computer readable non-transitory medium containingprogramming instructions that, when executed, will cause the processorto perform certain functions. The processor receives a patient imagecomprising at least a body of a patient and sensor data captured by oneor more augmented reality (AR) sensors. The processor also receives amedical image, generates a representation of the medical image, andperforms coregistration between the patient image and the representationof the medical image to generate a transformation matrix. The processoralso superimposes the representation of the medical image onto thepatient image based on the transformation matrix to form a superimposedimage, and displays the superimposed image on the display.

Alternatively, and/or additionally, the system performs thecoregistration manually by: displaying the patient image on the display,displaying the representation of the medical image on the display,receiving a user input to move the representation of the medical imagerelative to the patient image on the display, and generating thetransformation matrix based on the relative location between therepresentation of the medical image and the patient image.

Alternatively, and/or additionally, the system perform thecoregistration automatically by: extracting one or more features fromthe representation of the medical image, generating volumetric databased on the sensor data, and generating the transformation matrix basedon the one or more features and the volumetric data.

Alternatively, and/or additionally, the one or more features include afiducial, and the sensor data comprises information about the fiducial.

Alternatively, and/or additionally, the fiducial is a skin or anexternal surface of the patient image.

Alternatively, and/or additionally, the fiducial is a deep structure ofthe body of the patient or a marker placed on the body of the patient.

Alternatively, and/or additionally, the fiducial is an artery or septaldivide between compartments of the body of the patient.

Alternatively, and/or additionally, at least one of the one or more ARsensors includes a camera, a three-dimensional (3D) scanning device, oran ultrasound device.

Alternatively, and/or additionally, the system is configured todetermine a change of the body of the patient.

Alternatively, and/or additionally, the system is configured todetermine a movement of the body of the patient. If the movement of thebody of the patient has exceeded a threshold, the system updates thetransformation matrix to generate an updated transformation matrix.

Alternatively, and/or additionally, the system updates thetransformation matrix by: determining information about the movement ofthe body; and updating the transformation matrix based on theinformation about the movement of the body.

Alternatively, and/or additionally, the information about the movementof the body comprises a position change of the body from a previousposition.

Alternatively, and/or additionally, the system updates thetransformation matrix by: receiving an updated patient image, andperforming coregistration between the updated patient image and therepresentation of the medical image to generate the updatedtransformation matrix.

Alternatively, and/or additionally, the representation of the medicalimage is a hologram.

Alternatively, and/or additionally, the system updates therepresentation of the medical image based on the information about themovement of the body.

Alternatively, and/or additionally, the body of the patient comprises atleast one of a nerve, an artery, or an internal organ.

Alternatively, and/or additionally, the system determines a change ofthe body of the patient in size. If the change of the size of the bodyof the patient has exceeded a threshold, the system updates thetransformation matrix to generate an updated transformation matrix.

Alternatively, and/or additionally, the system assesses a function of aheart, a lung or an internal organ of the patient, or assesses a brainedema or blood loss.

Alternatively, and/or additionally, the patient image includes asurgeon's hand or a surgical instrument in the surgeon's hand.

Alternatively, and/or additionally, the system superimpose the surgeon'shand or the surgical instrument on the medical image. Alternatively,and/or additionally, the system determines a change in a position orshape of the surgical instrument, and superimposes the surgicalinstrument on the medical image based on the change in the position orthe shape of the surgical instrument.

Alternatively, and/or additionally, the display is a display of an ARdevice.

Alternatively, and/or additionally, the display is configured to rendera hologram.

Alternatively, and/or additionally, the display is configured to displaya 3D binocular vision.

Alternatively, and/or additionally, the display is configured to displayan image of the patient image by a scaling factor, the scaling factor isequal or less than one.

In another aspect of the disclosure, a method in a surgical navigationincludes: receiving a patient image comprising at least a body of apatient and sensor data captured by one or more augmented reality (AR)sensors; receiving a medical image; generating a representation of themedical image; performing coregistration between the patient image andthe representation of the medical image to generate a transformationmatrix; superimposing the representation of the medical image onto thepatient image based on the transformation matrix to form a superimposedimage; and displaying the superimposed image on the display.

Alternatively, and/or additionally, the method performs thecoregistration by: extracting one or more features from therepresentation of the medical image; generating volumetric data based onthe sensor data; and generating the transformation matrix based on theone or more features and the volumetric data.

Alternatively, and/or additionally, the method also includes determininga movement of the body of the patient. If the movement of the body ofthe patient has exceeded a threshold, the method updates thetransformation matrix to generate an updated transformation matrix.

Alternatively, and/or additionally, the method also includes:determining a movement of a surgical instrument in the patient image;and superimposing the surgical instrument on the medical image based onthe movement of the surgical instrument.

In addition, certain terms used in the present disclosure, including thespecification, drawings and claims thereof, can be used synonymously incertain instances, including, but not limited to, for example, data andinformation. It should be understood that, while these words, and/orother words that can be synonymous to one another, can be usedsynonymously herein, that there can be instances when such words can beintended to not be used synonymously. Further, to the extent that theprior art knowledge has not been explicitly incorporated by referenceherein above, it is explicitly incorporated herein in its entirety. Allpublications referenced are incorporated herein by reference in theirentireties.

What is claimed is:
 1. A system comprising: a processor; a display; anda computer readable non-transitory medium containing programminginstructions that, when executed, will cause the processor to: receive apatient image comprising at least a body of a patient and sensor datacaptured by one or more augmented reality (AR) sensors; receive amedical image; generate a representation of the medical image; performcoregistration between the patient image and the representation of themedical image to generate a transformation matrix; and superimpose therepresentation of the medical image onto the patient image based on thetransformation matrix to form a superimposed image; and display thesuperimposed image on the display.
 2. The system of claim 1, wherein theprogramming instructions for performing the coregistration compriseprogramming instructions configured to: display the patient image on thedisplay; display the representation of the medical image on the display;receive a user input to move the representation of the medical image toa location relative to the patient image on the display; and generatethe transformation matrix based on the relative location between therepresentation of the medical image and the patient image.
 3. The systemof claim 1, wherein the programming instructions for performing thecoregistration comprise programming instructions configured to generatethe transformation matrix automatically by: extracting one or morefeatures from the representation of the medical image; generatingvolumetric data based on the sensor data; and generating thetransformation matrix based on the one or more features and thevolumetric data.
 4. The system of claim 3, wherein: the one or morefeatures include a fiducial; and the sensor data comprises informationabout the fiducial.
 5. The system of claim 4, wherein the fiducial is askin or an external surface of the patient image.
 6. The system of claim4, wherein the fiducial is a deep structure of the body of the patientor a marker placed on the body of the patient.
 7. The system of claim 4,wherein the fiducial is an artery or septal divide between compartmentsof the body of the patient.
 8. The system of claim 1, wherein at leastone of the one or more AR sensors includes a camera, a three-dimensional(3D) scanning device, or an ultrasound device.
 9. The system of claim 1further comprising additional programming instructions configured todetermine a change of the body of the patient.
 10. The system of claim9, wherein the additional programming instructions comprise programminginstructions configured to: determine a movement of the body of thepatient; and if the movement of the body of the patient has exceeded athreshold, update the transformation matrix to generate an updatedtransformation matrix.
 11. The system of claim 10, wherein theprogramming instructions for updating the transformation matrix compriseprogramming instructions configured to: determine information about themovement of the body; and update the transformation matrix based on theinformation about the movement of the body.
 12. The system of claim 11,wherein the information about the movement of the body comprises aposition change of the body from a previous position.
 13. The system ofclaim 10, wherein the programming instructions for updating thetransformation matrix comprise programming instructions configured to:receive an updated patient image; and perform coregistration between theupdated patient image and the representation of the medical image togenerate the updated transformation matrix.
 14. The system of claim 1,wherein the representation of the medical image is a hologram.
 15. Thesystem of claim 11 further comprising additional programminginstructions configured to: update the representation of the medicalimage based on the information about the movement of the body.
 16. Thesystem of claim 1, wherein the body of the patient comprises at leastone of a nerve, an artery, or an internal organ.
 17. The system of claim9, wherein the additional programming instructions comprise programminginstructions configured to: determine a change of the body of thepatient in size; and if the change of the size of the body of thepatient has exceeded a threshold, update the transformation matrix togenerate an updated transformation matrix.
 18. The system of claim 9further comprising additional programming instructions configured to:assess a function of a heart, a lung or an internal organ of thepatient; or assess a brain edema or blood loss.
 19. The system of claim1, wherein the patient image includes a surgeon's hand or a surgicalinstrument in the surgeon's hand.
 20. The system of claim 19 furthercomprising additional programming instructions configured to superimposethe surgeon's hand or the surgical instrument on the medical image. 21.The system of claim 20 further comprising additional programminginstructions configured to: determine a change in a position or shape ofthe surgical instrument; and superimpose the surgical instrument on themedical image based on the change in the position or the shape of thesurgical instrument.
 22. The system of claim 1, wherein the display is adisplay of an AR device.
 23. The system of claim 22, wherein the displayis configured to render a hologram.
 24. The system of claim 22, whereinthe display is configured to display a 3D binocular vision.
 25. Thesystem of claim 22, wherein the display is further configured to displayan image of the patient image by a scaling factor, the scaling factor isequal or less than one.
 26. A method comprising: receiving a patientimage comprising at least a body of a patient and sensor data capturedby one or more augmented reality (AR) sensors; receiving a medicalimage; generating a representation of the medical image; performingcoregistration between the patient image and the representation of themedical image to generate a transformation matrix; superimposing therepresentation of the medical image onto the patient image based on thetransformation matrix to form a superimposed image; and displaying thesuperimposed image on the display.
 27. The method of claim 26, whereinperforming the coregistration comprises: extracting one or more featuresfrom the representation of the medical image; generating volumetric databased on the sensor data; and generating the transformation matrix basedon the one or more features and the volumetric data.
 28. The method ofclaim 26 further comprising: determining a movement of the body of thepatient; and if the movement of the body of the patient has exceeded athreshold, updating the transformation matrix to generate an updatedtransformation matrix.
 29. The method of claim 26 further comprising:determining a movement of a surgical instrument in the patient image;and superimposing the surgical instrument on the medical image based onthe movement of the surgical instrument.